Method and device for modifying the affective visual information in the field of vision of an user

ABSTRACT

Disclosed is to a method and to a device for modifying the affective visual information in the field of vision of a user of the device. The device includes at least one image sensor and at least one display. The method includes the following steps: Detecting an image in the field of vision of the device using an image sensor, carrying out a face recognition using the detected image for recognizing at least one face, determining the position of the eyes and mouth of the detected face, calculating a superimposition area in the display of the device in accordance with the determined positions of the eyes and mouth, superimposing the field of vision of a user of the device with alternative image data in the calculated superimposition of the display.

The invention relates to a method and a device for changing theaffective visual information in the field of vision, in particular inthe field of view, of a user. The device has at least one image sensorand at least one display as well as a processing unit that is connectedto the image sensor and the display, the image sensor being set up toacquire an image of a visual field of the device, the display being setup to superimpose the field of vision of a user in a superimpositionregion of the display with alternative image data.

Here, the field of vision of the user is that solid angle or region ofexternal space from which at least one eye of the user can receive andevaluate light pulses. It is composed of the monocular fields of visionof the two eyes. In comparison, the field of view here is that regionthat contains in the external space all visual objects that can becentrally fixed in succession with the eyes with the head and body heldat rest; this region is also more precisely called the monocular fieldof view. The visual field finally designates the region in the angle ofview of an optical instrument or an optical device, i.e., the region ofexternal space that can be viewed by the device (with one eye) or thatcan be imaged (for example, with an image sensor).

Devices and methods are known with which insertion of additionalinformation, i.e., which goes beyond the reproduction of the vicinity,in the field of vision of a user is possible. These devices and methodsare combined under the term “Augmented Reality” (AR). For example, US2011/0214082 A1 shows such a device in the manner of a head-mountedviewing device with transparent display that can be called “AR glasses”for short. This device comprises a camera for acquiring images of thevicinity, in particular a visual field of the device. These images canbe processed, and the results of the processing can be used for choosingthe inserted information. These AR devices are generally designed tosupply the user with information that augments the information that canbe directly optically perceived in the vicinity (“to augment” means toamplify, to expand) and differs from it.

In another technical connection, in video conferencing systems, changingthe information that can be directly optically perceived by the user bycross-fading of certain image regions is known. U.S. Pat. No. 9,060,095B2 describes a method in which the face of a participant in a videoconference is replaced by a replacement image, for example a schematicface or a foreign face. Thus, the objective is to hide the identity ofthe participant from the other participants of the video conference. Thecross-faded face of the pertinent participant cannot be viewed orrecognized by the other participants. This protection of the identityis, of course, not transferable to AR applications, because a wearer ofan AR device as described above can take it off in order to see theactual face of his interlocutor. Accordingly, it would contradict thepurpose of the method to intend to use it for individuals in thevicinity of a wearer of an AR device.

One object of the invention is to propose a method and a device withwhich the affective perception of a user can be adapted when viewing hisvicinity.

The method according to the invention for changing the affective visualinformation in the field of vision, in particular the field of view, ofa user of a device with at least one image sensor and at least onedisplay comprises the following steps: acquisition of an image of avisual field of the device with the image sensor, facial recognitionwith the acquired image for recognizing at least one face, determiningthe positions of eyes and mouth of the recognized face, computing asuperimposition region in the display of the device depending on thedetermined positions of eyes and mouth, and superimposition of the fieldof vision of a user of the device in the computed superimposition regionof the display with alternative image data. The visual field of thedevice is generally an extract from the field of vision of the user. Itdoes not necessarily correspond to the region imaged by the imagesensor; in particular, the imaged region can comprise the visual fieldof the device and can go beyond it. The image sensor can be, forexample, part of a visual field camera that has a larger visual fieldthan the device. The device can be, for example, glasses, a camera,binoculars, a sight, or another optical device. The image sensor and thedisplay are part of the same device and preferably have a fixed relativearrangement. In facial recognition, the number and the positions offaces in the image acquired with the image sensor are recognized. Thisfacial recognition can proceed with conventional means and algorithms.When computing the superimposition region, essentially the position anddimensions of the superimposition region in the display are computed.The superimposition of the field of vision or field of view of the useris achieved in particular by insertion into a visual field of the devicewith the display.

The device of the initially-cited type according to the invention forchanging the affective visual information in the field of vision, inparticular in the field of view, of a user is characterized in that theprocessing unit is set up to carry out facial recognition with an imageacquired by an image sensor, to determine positions of eyes and mouth ofa face recognized in the facial recognition, and to compute asuperimposition region in the display for superimposing the field ofvision depending on the determined positions of eyes and mouth. Thedevice is preferably an AR device. The content of the display iscontrolled using data that has been acquired by the image sensor, thedisplay and image sensor being located essentially at the same site,specifically as part of the same device.

By superimposing the field of vision of the user in a superimpositionregion that is dependent on the positions of the eyes and mouth ofrecognized faces, specific superimposition of these faces can beachieved, which can be used for regulation of the visual perception ofaffects on individuals in the field of vision or field of view of theuser.

It has been ascertained that this regulation of the visual perception ofaffects has a major influence on the capacity for empathy of theindividuals supported in this way. For example, in individuals withADHD, a positive effect of this regulation on self-confidence and ongood self-effectiveness has been ascertained. Moreover, stress reactionin social interaction can be controlled and attenuated.

Affective information in this connection is all signals that express anemotion of an individual. Depending on the psychological state of theindividual, this information comprises gradual expressions of actualemotions and expressions for hiding actual emotions (“affective facade”)or in the absence of specific emotions, expressions ofcognitively-generated emotions that are rated as expected or useful. Inthe same manner as the intensity of expression, the intensity ofperception for affective information is individually different anddependent on the psychological state and also genetically disposed.

The superimposition of the field of vision preferably comprises arepresentation of the alternative image data in the superimpositionregion of the display.

Accordingly, the display can be a screen or a display, in particular atransparent display.

Preferably, the change in the affective visual information correspondsto a gradual diminishment or intensification of the affective visualinformation. That is, the affective visual information is preferably notcompletely erased or overwritten. The gradual diminishment orintensification can be achieved by, for example, local gradual change inthe transparency of a display. For example, the transparency can bereduced in one superimposition region to a suitable percentage, forexample to 50%.

Accordingly, this method is preferably a method for in particulargradual reduction/diminishment/disruption or intensification/enhancementof the affective visual information in the field of vision of a user ofthis device.

Advantageously, there can be a fluid transition at the boundary or edgeof the superimposition region. For example, a gradual course of the edgecan be achieved by means of linear variation of the transparency. Indoing so, the transparency in a transition region of the superimpositionregion on the boundary of the superimposition region in the directionperpendicular to the boundary of the superimposition region can increasefrom within the superimposition region to outside of the superimpositionregion. The transition region corresponds essentially to a strip alongthe boundary of the superimposition region. The width of the transitionregion is preferably matched to the width of the superimposition region;it is, for example, between 5 and 30% of the width of thesuperimposition region, preferably between 15 and 25%. The variation ofthe transparency is preferably linear over the width of the transitionregion and varies preferably between complete transparency (or maximumtransparency of the display used) on the boundary of the superimpositionregion and the selected transparency outside of the transition region(i.e., in the remainder of the superimposition region), for examplecomplete opacity (or minimum transparency of the display). The fluidtransition achieved by the transition region reduces any irritation ofthe user caused by the superimposition.

Preferably, the device has two image sensors that are located at a fixeddistance from one another. This enables the determination of thedistance to a recognized face, as a result of which more accuratecomputation of the dimensions of the superimposition region in thedisplay is enabled.

Especially preferably, the device comprises augmented reality glasses.Suitable glasses are offered by, for example, Osterhout Design Group,153 Townsend Street, Suite 570 San Francisco, Calif. 94107, inparticular its products “R-6” or “R-7 Smartglasses System.”

The superimposition region can correspond essentially to a trianglespread out between the positions of the two eyes and the mouth. That is,the computation of the superimposition region comprises a connection ofthe positions and determination of the image points of the display thatare located in the encompassed triangle(s). Alternatively, othergeometrical shapes can also be used whose size is scaled according tothe indicated triangle. For example, a symmetrical trapezoid can be usedin which the wider base side is located in the region of the eyepositions. Optionally, the wider base side can be lengthened to the topby a rectangle. It has been ascertained that in the region of theeye-eye-mouth triangle, the largest affective amount of data istransported, in particular by the mimetic musculature. A comparativelysmaller part of the affective information is imparted in conventionalinterpersonal interaction by the voice and body language.Superimposition of information in this region of the field of visiontherefore results in a reduction of the affective information that hasbeen perceived by the user; this can be used for regulation. Thus, theaffective information perceived by users with unusually strong affectiveperception can be reduced to a level that corresponds to averageeffective perception.

On the other hand, the superimposition region can also compriseessentially the entire field of view outside of a triangle spread outbetween the positions of the two eyes and the mouth. That is, in thiscase, the computation of the superimposition region comprises aconnection of the positions and determination of the image points of thedisplay that are located outside of the encompassed triangle ortriangles. A superimposition of the region outside of the so-calledeye-eye-mouth triangles enables emphasis of the eye-eye-mouth triangleand a deflection of attention to these triangles, for example by dimmingthe remaining superimposed region. The superimposition then results inan amplification of the affective information that has been perceived bythe user. Thus, the affective information perceived by users withunusually weak affective perception can be raised to a level thatcorresponds to average affective perception.

The processing unit of the device in question can preferably have facialrecognition software. By means of the facial recognition software, onthe one hand, faces can be recognized in the acquired image data, andthe positions of eyes and mouth of the individual recognized faces canbe determined.

Alternative image data can preferably correspond to altered imaging ofthe visual field of the device in the superimposition region, the changein the imaging encompassing a change in brightness, a reduction in imageresolution, a change in delineation (also called sharpness) and/or incontrast, color marking (for example, by means of a frame and/or colortinting), and/or manipulation of facial traits. The indicated changes inimaging contain a part of the original information in thesuperimposition region. That is, not all optical information in thesuperimposition region is replaced. In this way, the superimposition isless conspicuous to the user, so that his attention is notunintentionally diverted to the superimposition region. One suchdiversion that appears unintentional for the vicinity could, of course,have an unnatural and irritating effect on a party other than the user;this can be avoided with the proposed changes. Essentially skin-coloredtinting in the superimposition region has proven particularly favorable.Especially minor irritation of the perception with, moreover, effectivedisruption of affective visual information can be achieved with apartially transparent, preferably horizontal, strip pattern in thesuperimposition region. The number of strips within a recognized facialregion is preferably greater than four and less than 50. The color ofthe strip pattern can vary between the color of the recognized facialfolds (dark) and relatively flat facial surfaces (bright) and thus canbe matched to the face recognized at any one time.

Furthermore, within the scope of this method, it is advantageous toprovide for acquisition of the signal of at least one biofeedbacksensor, the superimposition being activated as a function of the signalof the biofeedback sensor. Preferably, superimposition can be activatedonly at a stress level that has been determined by the biofeedbacksensor above a predefined threshold so that below the threshold, anunchanged perception of the vicinity is possible. The superimposition isactivated here preferably only for a fixed short time interval, afterwhose expiration the unchanged visual field can be perceived again. Thebiofeedback sensor can be in particular a stress sensor, for example aphotoplethysmogram (PPG) or a sensor for measuring skin resistance, skintemperature, heart rate and/or the motion of the body. One suchbiofeedback sensor is, for example, the product “Q sensor” from thecompany Affectiva, 465 Waverley Oaks Road, Suite 320, Waltham, Mass.02452, United States, or the products “Embrace” and “E4” from thecompany Empatica Inc, 1 Broadway, 14th floor, Cambridge, Mass. 02142,United States.

Accordingly, the device in question can comprise a biofeedback sensor,the processing unit being set up and connected to the biofeedback sensorto activate the superimposition depending on a signal of the biofeedbacksensor.

In conjunction with the evaluation of a signal of a biofeedback sensor,it is especially advantageous if the extent of the change in the imagingis determined as a function of the signal of the stress sensor. Forexample, in this way, with an increasing stress level, stronger dimmingof the affective information can be undertaken and vice versa. In thisway, negative circulation, in which the perception of the other party ofincreased stress on the user contributes to affective information thatfurther increases the stress, can be broken through early. Moreover, thechange in the imaging can be limited to those situations in which it isnecessary or helpful (for example under stress or worry). Outside ofthese situations, the change in the affective information is minimizedso that a natural mutual perception is achieved.

Likewise, in conjunction with the acquisition of the signal of at leastone biofeedback sensor, within the scope of this method, the alternativeimage data can correspond to a reference recording of the superimposedfacial region. The reference recording can be stored at a fixed low ornormal stress level. Thus, the reference recording shows, for example, afacial expression with little visual affective information and/or withpositive or calming visual affective information. As soon as a highstress level has been ascertained, the reference recording is used forsuperimposition of the same face. For scaling and alignment of thereference recording, the eye-eye-mouth triangle that has been recognizedboth in the reference recording and also in the superimposition regionis used. By the superimposition, for example, of one facial region witha reference recording of the same facial region, an especially naturalimage and consequently especially low irritation of the user can beachieved.

In the case of devices in which depending on the eye position of theuser when using the device depends on the position of the eyes [sic], itis advantageous if this method, before computing the superimpositionregion, calls for calibration of the display of the device for a user,an eye distance and a distance between the eyes and the display beingdetermined, the ascertained eye distance and the ascertained distancewhen computing a superimposition region being taken into account in thedisplay. By calibration, the superimposition region can be moreaccurately computed, so that controlled superimposition of certainprecisely delineated regions in the visual field of the device isenabled. Without calibration, conversely the error of estimation of theeye distance and the distance to the display must be considered, forwhich reason a comparatively larger superimposition region must becomputed in order to reliably achieve the desired superimposition.

It has furthermore turned out to be advantageous within the scope of themethod in question to provide for triggering an optional usernotification when a coupled biofeedback sensor, in particular a coupledstress sensor, is activated (for example, by means of measurement ofpulse frequency, muscle tone, EEG measurement and/or skin conductionrate), i.e., when a signal of a coupled biofeedback sensor exceeds apredefined threshold, and/or when the facial recognition (for example inthe form of facial recognition software) detects an affective deflectionor resonance above a predefined threshold on a recognized face (i.e., onthe individual opposite) and/or when voice recognition recognizes avoice of an individual that has been stored beforehand as a stressor ina database. One such user notice or notification can be achieved with ahaptic signal (for example, vibration), an acoustic signal (for example,alarm) or optical signal (for example, blinking display), which aredelivered from a pulse generator.

Correspondingly, the device in question can preferably comprise anotification apparatus that is set up to deliver an optical, acoustic orhaptic signal, the processing unit being set up and connected to thenotification apparatus to trigger an optical, acoustic or haptic signalwhen a specific affective pattern and/or an individual assigned to aspecific affect is recognized. Here, a specific affective pattern isdefined both as an affective reaction of the user of the device that canbe ascertained, for example, by means of a biofeedback sensor, and alsoan affective reaction of the other party, i.e., on at least onerecognized face that can be ascertained, for example, within the scopeof facial recognition using the facial expression. An individualassigned to a specific affect can be ascertained, for example, withinthe scope of additional voice recognition in conjunction with a databaseof individuals stored beforehand as a stressor or their voicecharacteristics.

The notification enables a user with little affective perceptionstrength to realize the affective information transported by anotherparty or to consciously perceive and react to it accordingly. When usingthe device or the method over a longer time interval, the attentivenessto affective information is trained by the notifications, and, in thisway, the natural capability for empathy of the user is enhanced.

Preferably, the superimposition can be activated as a function of theaforementioned voice recognition, the superimposition only beingactivated when the voice of an individual filed in a database (ofstressors) that is designed for this purpose is recognized.

In another preferred embodiment, the superimposition can be activated asa function of the identification of a recognized face, thesuperimposition only being activated when the face of an individualfiled in a database (of stressors) that is designed for this purpose isrecognized and identified.

Preferably, within the scope of the invention, at least three operatingmodes can be selected by the user:

-   1) a permanently activated superimposition, i.e., of each recognized    face;-   2) an individual-specific superimposition, the superimposition being    activated as a function of a voice recognition and/or facial    recognition;-   3) activation of the superimposition by means of a biofeedback    sensor (or stress sensor), i.e., the superimposition being activated    as a function of the stress level of the user.

In addition to the above-described applications, the invention alsorelates to a method for measuring the sensitivity of a subject foraffective visual information, in particular for sole or supplementarydiagnostics of anxiety conditions and depressions in general andespecially diagnostics of ADHD or diseases from the autistic spectrum.This method comprises the above-described method for changing affectivevisual information, the subject being the user of the device accordingto the invention and the superimposition being varied over time (forexample, deactivated in alternation for a few minutes, then activatedfor a few minutes), and at the same time the affective reaction of thesubject being determined with a stress sensor. A measure of thesensitivity for affective visual information or at least insensitivitycan be ascertained from the time correlation between the timecharacteristic of a stress level measured using the stress sensor andthe time characteristic of the superimposition.

Moreover, the invention relates to the use of a method or a device ofthe initially-cited type with the above-described preferred embodimentsfor preventing social stress situations, for precluding anxietyconditions and depressions, or for stress reduction among a portion ofdepressive and anxious individuals. Depending on the variance of affectperception, in depressive conditions as well as in anxiety conditions,in the respective causality, (a) an affect-dependent form and (b) a formthat is hardly affect-dependent to not affect-dependent can be found inthe origination of these conditions. In the case of (a), a strongperception of the affects in the face of a viewed individual, such as,for example, in ADHD (ICD-10 Code F90.-), corresponds to the ideal type,and in the case of (b), a slight to deficient perception in this region,such as in, for example, autism (Icd-10 Code F84.-), corresponds to theideal type. Concordant with the previous tests in the ordination of theinventor, the “Lazar Effect,” therefore, a stress reduction, can beascertained by influencing the affect perception (in the sense of areduction of the information content in the corresponding visualregion), for a portion of depressive and anxious patients (ICD-10 CodeF3-.- and F4-.-), as well as for all ADHD patients. For these groups ofindividuals, the use of this invention is therefore particularlyadvantageous. This, while another portion of depressive and anxiouspatients does not experience any stress reduction at all, thisexclusively for affected individuals with little to no automated affectperception relative to other individuals [sic]. In two patients withconditions from the autistic spectrum, both suffer from Asperger'ssyndrome; likewise, no stress reduction at all could be ascertained inthe use of this invention. This corresponds exactly to the hypothesisregarding the expected Lazar effect and its meaning for the patientgroup described under (a). Since the perception capacity of healthyindividuals can likewise be divided into two groups corresponding to theideal type in each case, and it can occur as a result ofmisunderstanding with respect to the semantics of emotional perceptionor affect perception because most individuals assume that the otherparty has the same affect perception, these two terms are defined stillmore precisely below: The group corresponding to (a) here means at thesame time affect-dependent, the automated perception of affects (inparticular of muscle contractions in the face) that is independent ofintellectual control and consciousness in the case of viewedindividuals. This processing process is not present in group (b) and inindividuals from the healthy range that correspond to this ideal type.It is therefore in most cases not understood by the latter for lack oftheir own structural conditions. The latter is based on a mechanism ofperception of other individuals among psychiatric laymen, often alsocalled general human knowledge. This is based on the false assumptionthat other individuals function and perceive according to the sameprinciples. The group corresponding to (b) perceives affects solely viaa deliberate and therefore conscious act that immediately stops whenattention is withdrawn from the viewed individual. This is comparable tothe understanding of a non-native language that is only understood whenattention is focused on it. The affect-dependent perception converselymeans the constant perception of the affects of the other party,corresponding to automatic hearing and understanding of the nativelanguage.

The invention is explained still further below using an especiallypreferred exemplary embodiment to which, however, it is not to belimited with reference to the drawings. Here, in detail:

FIG. 1 schematically shows a part of a device according to the inventionin the form of AR glasses;

FIG. 2 schematically shows a block diagram of the functional elements ofa device according to the invention;

FIG. 3 schematically shows an illustration of the computation of asuperimposition region;

FIG. 4 shows one application example of the method according to theinvention;

FIG. 5 schematically shows an elliptical superimposition region with atransition region with transparency that rises linearly to the outside;

FIGS. 6A, 6B, 6C and 6D show four application examples for an ellipticalsuperimposition region with a transition region with transparency thatrises linearly to the outside for respectively different widths of thetransition region;

FIG. 7 schematically shows a flow chart of a method for using areference recording for superimposition of a facial region.

FIG. 1 shows a simplified representation of AR glasses 1, such as, forexample, the “R7 Smart Glasses” offered by the Osterhout Design Group(ODG). The lenses of the AR glasses 1 are two displays 2, which are eachmade as a transparent display (look-through display). The graphicsurface displayed on these displays is generated by a processing unit(not shown), which is operated with the operating system (OS) Androidwith the OS expansion “ReticleOS” (likewise from ODG).

A camera 3 integrated into the AR glasses 1 comprises an image sensorand delivers an image of the vicinity. The visual field of the camera 3covers essentially the half-space in front of the lenses of the glassesand is thus larger than the visual field of the AR glasses 1. The visualfield of the AR glasses 1 corresponds to that part of the field ofvision of a user that is being perceived by the lenses of the glasses.The image of the vicinity that has been acquired by the camera 3 istransmitted via a cable in the frame 4 of the glasses and a connectingline 5 to a processing unit 6 (see FIG. 2) and is continuously analyzedthere by facial recognition software, which outputs the coordinates ofthe face or faces in the vicinity.

An application program that runs on the processing unit 6 firstdetermines from the pre-calibrated eye distance the region of therecognized face. Then, from the coordinates and the parametersestablished within the scope of calibration on the physiognomy of theindividual wearing the glasses (eye distance, distance between lenses ofthe glasses and eye) for each lens of the glasses, it continuouslycomputes the coordinates of the point of intersection of the viewingaxis to the face of the other party with the lenses of the glasses,therefore that site in each lens of the glasses that is suitable tocover the eye-eye-mouth triangle (AAM triangle) of the other party.These data are in turn transferred to the API (programming interface) ofthe ReticleOS that inserts a preestablished image or pattern at thecorresponding site in the lenses of the glasses.

The structure of the application program allows the merging of softwareexpansions. They include, for example, expansions for continuouschanging of the pattern or image used for cross-fading. The applicationprogram can thus be expanded, for example, to the extent that theinserted image is the face that has been photographed with the camera,recognized and output by the facial recognition software, and in whichthe facial features have been manipulated in a dedicated manner.

As shown in FIG. 2, the processing unit 6 can be optionally connected tosensors 7 and input elements 8. The sensors 7 can comprise a biofeedbacksensor for measuring stress parameters. The measured parameters can beused to evaluate the data of the facial recognition software and assignthem to a quality. For this purpose, for example, measurements of skinresistance, skin temperature, heart rate, or the movement of the bodycan be used as stress parameters.

The processing unit 6 together with the AR glasses 1 forms a deviceaccording to the invention. The latter is, moreover, connected to apower management unit 9 and a rechargeable battery 10.

The superimposition of the field of vision by means of the AR glasses 1can be used fundamentally for two types of influence: First, a reductionof the affective information content by a generic change in the imagedata (brightness, contrast, portrayal, etc.) or, second, asuperimposition of the face with another face or a face of the sameindividual with a modified affective information content, in particularsmaller negative emotional content than the original image. The lattercan be photographed prior to that by the camera 3.

The degree of acceptable influence on communication is limited by thecommunicative plausibility and is defined as given adequacy based on thepsychiatric classification of the psychopathological status.

When this value is exceeded, a rapid rise of the affective informationtransported by the observed other party takes place, and as a result, byhis perception for the user/individual wearing the glasses, a strongdeflection of the stress sensors for the latter takes place. Moreover,the latter leads to a switching-over of the emotional perception fromunintentional to intentional perception and to a reduction in adequacy.

It is one objective of this invention to control communicationunintentionally and slowly in the direction of positive expression andperception of emotions among the interlocutors without reaching orexceeding the threshold value of adequacy.

FIG. 3 illustrates the method steps for computing the superimpositionregion using AR glasses 1 according to FIG. 1.

In the illustrated exemplary situation, the eyes AL (left eye) and AR(right eye) of an observing subject (the user and wearer of the ARglasses) are schematically shown. The eyes AL, AR are shown respectivelyby the opening of the pupil and the round eyeball with the retina on itsposterior inner side. The coordinates of the points AL and AR correspondto those of the center of the respective eye. The centers of the pupilsare marked with PL and PR for the pupil of the left and right eyerespectively. The distance a_(PD) corresponds to the measured pupildistance and stands for the distance between the centers of the two eyesAL, AR.

The subject wears AR glasses with two displays 2 as lenses of theglasses that are implemented as transparent displays. DL designates thelens of the glasses or display that is the left one from the subject'sviewpoint and DR the right one. The points DL and DR are defined as thecenter of the respective display 2. Mounted on the glasses are twocameras 11, 12, which are each shown as a camera obscura, consisting ofa diaphragm 13 and a CCD image sensor 14. The point C (for camera) isdefined in the center of the diaphragm.

The coordinates x and y relate to the coordinate system whose origin isat the center of the image sensor 14 or in the left or right display DL,DR. Here, x is the horizontal coordinate, and y is the verticalcoordinate. The indices indicate which coordinate system it is. Thus,the coordinates x_(DL), y_(DL) relate to the coordinate system with theorigin point DL in space, and the coordinates x_(DR), y_(DR) relate tothe one with the origin point DR, i.e., the center of the left and rightdisplay 2. The coordinates x_(C)and y_(C) relate to that coordinatesystem whose origin is at the center of the image sensor 14. Thedisplays 2 and the image sensor 14 of the camera 11 are parallel to thex-y plane in space. The z-coordinate that is perpendicular theretocorresponds to the direction of the camera. The image plane of FIG. 3corresponds to the x-z plane in space.

The other party whose facial features are being read is called theobject (O). The eyes of the face of the object facing theobserver/subject (S) are designated OL for the left eye from theviewpoint of the subject and OR for the right eye. The mouth isdesignated OM, the center of the face OC. Distances in the X directionare labeled a, those in the Y direction d, and those in the Z directionb. This choice is based on the conventional side designations of thetriangles computed below. The indices of these lengths indicate the endpoints of the pertinent line segments.

The drawing according to FIG. 3 represents the projection of the linesegments in space onto the x-z plane. Let f be the distance of thediaphragm 13 to the image sensor 14, which is known by the knowledge ofthe camera used. At this point, let b_(c) be the still unknown distanceof the camera 11 to the object (O). The image on the image sensor 14yields the distance of the image of the left eye OL of the object on theimage sensor 14 to the central axis x_(C,OL) as well as the distance ofthe image of the right eye OR of the object on the image sensor 14 tothe central axis x_(C,OR). The beam triangles to the right from (infront of) the diaphragm 13 to the object and to the left from (behind)the diaphragm 13 to the image sensor 14 are similar. Therefore, thefollowing applies:

$\begin{matrix}{\frac{a_{C,{OL}}}{b_{C}} = {\frac{x_{C,{OL}}}{f} = {\tan \left( \alpha_{C,{OL}} \right)}}} & (1)\end{matrix}$

as well as

$\begin{matrix}{\frac{a_{C,{OR}}}{b_{C}} = {\frac{x_{C,{OL}}}{f} = {\tan \left( \alpha_{C,{OL}} \right)}}} & (2)\end{matrix}$

At this point, the eye distance of the object 1_(O,PD) (or 1 for short)with an average value of 65 mm is assumed, and

a _(C,OL) =a _(C,OC) −l/2

as well as

a _(C,OR) =a _(C,OC) +l/2

This yields an equation system of two equations and two unknowns:

$\begin{matrix}{\frac{a_{C,{OC}} - \frac{l}{2}}{b_{C}} = \frac{x_{C,{OL}}}{f}} & (3)\end{matrix}$

as well as

$\begin{matrix}{\frac{a_{C,{OC}} + \frac{l}{2}}{b_{C}} = \frac{x_{C,{OR}}}{f}} & (4)\end{matrix}$

From the latter, the distance of the camera to the object b_(c) and thedistance a_(c,oc) of the center of the face OC of the object from thecentral axis of the camera 11 can be computed.

The center of the darkening of the display at this point follows from:

$\begin{matrix}{\frac{a_{C,{OC}} - a_{C,{AR}}}{b_{C} + b_{A\; C}} = {\frac{x_{DR}}{b_{AD}} = {\tan \left( \alpha_{{AR},{OC}} \right)}}} & (5)\end{matrix}$

and analogously for the left display:

$\begin{matrix}{\frac{a_{C,{OC}} - a_{C,{AR}} - a_{PD}}{b_{C} + b_{A\; C}} = {\frac{x_{DL}}{b_{AD}} = {\tan \left( \alpha_{{AL},{OC}} \right)}}} & (6)\end{matrix}$

Here, it has been assumed that the coordinate origin of the display islocated on the same axis as the center of the eye. If this is not thecase, the coordinates are to be shifted accordingly. In practice, itwill be the reverse: In the calibration phase, the darkening point canbe manually shifted. The center of the eye is then computed from x_(DR),and the coordinate transformation is approximated iteratively from allmanual interventions.

For the transformation of the y coordinate of the image sensor 14 ontothe y coordinate of the display, let d_(C,OA) be the projection of theline segment from the camera diaphragm to the eyes OR, OL of the otherparty/object onto the y-z plane and d_(C,OM) the projection of the linesegment from the diaphragm 13 to the mouth (OM) of the otherparty/object onto this plane, and let y_(C,OA)=y_(C,OR)=y_(C,OL), andy_(C,OM) be they coordinates of the image of these points on the imagesensor 14 of the camera 11:

$\begin{matrix}{\frac{d_{C,{OA}}}{b_{C}} = {\frac{y_{C,{OA}}}{f} = {\tan \left( \alpha_{C,{OA}} \right)}}} & (7)\end{matrix}$

as well as

$\begin{matrix}{\frac{d_{C,{OM}}}{b_{C}} = {\frac{y_{C,{OM}}}{f} = {\tan \left( \alpha_{C,{OM}} \right)}}} & (8)\end{matrix}$

Since b_(c) is already known from equations (3) and (4), d_(C,OM) andd_(c),_(oA) can be computed from this equation system. Analogously toequation (5), at this point, the y coordinate on the two displays can becomputed from the distances (determined in the calibration) between thecamera and eye d_(C,A) on the Y-Z plane:

$\begin{matrix}{\frac{d_{C,{OA}} - d_{C,A}}{b_{C} + b_{A\;,C}} = {\frac{y_{D,A}}{b_{A,D}} = {\tan \left( \alpha_{A,{OA}} \right)}}} & (9)\end{matrix}$

as well as

$\begin{matrix}{\frac{d_{C,{OM}} - d_{C,A}}{b_{C} + b_{A,C}} = {\frac{y_{D,M}}{b_{A,D}} = {\tan \left( \alpha_{A,{OM}} \right)}}} & (10)\end{matrix}$

If the origin of the coordinates of the display is not at the height ofthe center of the eye, as assumed here, the coordinates must again beshifted accordingly.

In the transformation of the y coordinates, the corner points y_(D,A),y_(D,M) of the superimposition region and thus also its dimensions havealready been determined. This can take place analogously in thetransformation of the x-coordinates, which, however, has been omittedfor reasons of clarity. In doing so, exactly as in the computation ofthe central point of the superimposition region, the eye center can beassumed, since the triangle (in the x-z plane) from the center of theeye to OL and OR completely superimposes the triangle from the pupil tothe OL.

In the computation of the extension of the superimposition region,moreover, that error that results from the different magnitudes of theeye distance l_(O,PD) of the other party/object can be taken intoaccount, i.e., the error of the assumed average value of 65 mm. An errorcomputation then yields the necessary increase of the extension of thesuperimposition region.

When using the second camera 12, two angles yield the acquired referencepoint at the object. Since the distance a_(CC) between the two camerasis known, the triangle between the first camera 11, the second camera12, and the object or its eyes OR, OL can be constructed using the WSWlaw or using the law of sines. The distance b_(C) of the cameras 11, 12to the object then corresponds to the height of the triangle. Thus, theestimation of the eye distance of the object is no longer relied upon,and the aforementioned miscalculation and the resulting enlargement ofthe superimposition region can be omitted.

For example, using the following equation:

$\begin{matrix}{\frac{a_{CC} - a_{COC} - \frac{l}{2}}{b_{C}} = \frac{x_{CLOR}}{f}} & (11)\end{matrix}$

the system of equations (3) and (4) can be solved even without knowledgeof the eye distance l_(o),_(PD).

FIG. 4 schematically shows one application example of the methodaccording to the invention using exemplary AR glasses 15. A field ofview camera 16 on the front of the AR glasses 15 acquires an image ofthe visual field 17 of the AR glasses 15 with an integrated imagesensor. An integrated processing unit carries out facial recognitionwith the acquired image for recognizing at least one face 18. In doingso, the position of the eyes and mouth of the recognized face 18 isdetermined, and then superimposition regions 19, 20 in the displays 21,22 of the AR glasses 15 are computed depending on the determinedpositions of the eyes and mouth (compare FIG. 3 and the descriptionabove in this respect). Finally, the field of vision of a user of thedevice is superimposed in the computed superimposition regions 19, 20 ofthe displays 21, 22 with alternative image data. The alternative imagedata are in each case partially transparent black surfaces that thusachieve local darkening of the field of vision in the superimpositionregions 19, 20 and in this way reduce the perceptible affectiveinformation of the face 18. Alternatively, the alternative image datacan be formed by in each case partially transparent skin-coloredsurfaces; this has a less irritating effect than black surfaces. Outsideof the superimposition regions 19, 20, the displays 21, 22 areuniformly, preferably completely, transparent comparable to aconventional optical glasses or sunglasses. The face 18 is drawn on thedisplays 21, 22 only for better understanding of the arrangement of thesuperimposition regions 19, 20, and thus represents the perspective ofthe user of the AR glasses 15, i.e., who is looking through the displays21, 22.

FIG. 5 schematically shows a view through one of the displays 21, 22with an elliptical superimposition region 23 for superimposition of aface 24. The superimposition region 23 encompasses a transition region25 with transparency that increases linearly to the outside. Outside ofthe transition region 25, the transparency of the superimposition region23 is essentially constant. The width of the transition region 25corresponds to roughly 10% of the width of the superimposition region 23at the widest point, measured parallel to a connecting line between therecognized eyes 26 of the face 24. The transition region 25 yields afluid transition between the superimposition region 23 and its vicinity,which has a less irritating effect for a user than a hard edge on theboundary of the superimposition region 23.

FIGS. 6A, 6B, 6C and 6D show four application examples for thetransition region that is described in conjunction with FIG. 5. Here, ineach case, a face with an elliptical, skin-colored superimpositionregion 23 is shown. The superimposition region 23 covers essentially theentire facial region between the ears and from the chin to the hairline.FIG. 6A shows the superimposition region 23 without the transitionregion 25; this corresponds to a width of the transition region 25 ofzero. FIGS. 6B, 6C and 6D show the superimposition region 23 with atransition region 25 according to FIG. 5. The width of the transitionregion 25 in FIG. 6B is 10% of the width of the superimposition region23; in FIG. 6C, it is 20% of the width of the superimposition region 23;and in FIG. 6D, it is 30% of the width of the superimposition region 23.In FIGS. 6C and 6D, the eyebrows of the face can be recognized. In FIG.6D, the external eye angles, the upper forehead region, and the mouthangle of the face can also be recognized. The width of the transitionregion can be used accordingly for adjusting the recognizable visualaffective information.

FIG. 7 shows a simplified flow chart of a method for use of a referencerecording for superimposition of a facial region according to oneexemplary embodiment of this method. Here, first of all in a loop 27, itis maintained until facial recognition 28 recognizes a face in the imagedata that have been acquired by the image sensor 14. As soon as a facehas been recognized, a stress level is determined (determination 29) bymeans of a stress sensor (for example, a biofeedback sensor). If thestress level is average or low (path 30), the image data in one regionof the recognized face are stored as a reference recording (storage 31).Together with the reference recording, the positions of the cornerpoints of the eye-eye-mouth triangle in the reference recording aredetermined and stored (determination and storage 32) in order toaccelerate later scaling and alignment of the reference recording. Ifthe stress level is still high (path 33), a current reference recordingis loaded (loading 34), which corresponds to the recognized face. Theassignment of the suitable reference recording takes place viabiometrics features of the acquired faces in a way that is known in theart. After loading 34, the reference recording is matched to the size,alignment and position of the recognized face by scaling, rotation andshifting and is then inserted as alternative image data into thedisplays 21, 22 (adaptation and insertion 35). Subsequently, it ischecked whether other faces have been recognized (checking 36). If no,the stress level is determined again, and the use of the superimpositionis re-evaluated, and optionally the adaptation is updated (loop 37). Ifyes, the method is repeated proceeding from the facial recognition 28for the next face (loop 38).

1. Method for changing the affective visual information in the field ofvision of a user of a device with at least one image sensor (14) and atleast one display (2), comprising the following steps: acquisition of animage of a visual field of the device with the image sensor (14), facialrecognition with the acquired image for recognizing at least one face,determination of the positions of eyes and mouth of the recognized face,computation of a superimposition region in the display (2) of the devicedepending on the determined positions of eyes and mouth, superimpositionof the field of vision of a user of the device in the computedsuperimposition region of the display (2) with alternative image data.2. Method according to claim 1, wherein the superimposition of the fieldof vision comprises a representation of the alternative image data inthe superimposition region of the display (2).
 3. Method according toclaim 1, wherein the superimposition region corresponds essentially to atriangle spread out between the positions of the two eyes and of themouth.
 4. Method according to claim 1, wherein the superimpositionregion encompasses essentially the entire field of view outside of atriangle spread out between the positions of the two eyes and the mouth.5. Method according to claim 1, wherein the alternative image datacorrespond to altered imaging of the visual field of the device in thesuperimposition region, the change in the imaging encompassing a changein brightness, a reduction in image resolution, a change in delineationand/or in contrast, color marking and/or manipulation of facial traits.6. Method according to claim 5, further comprising acquisition of thesignal of a least one biofeedback sensor, the superimposition beingactivated depending on the signal of the biofeedback sensor.
 7. Methodaccording to claim 6, wherein an extent of the change in the imaging isdetermined depending on the signal of the biofeedback sensor.
 8. Methodaccording to claim 1, further comprising, before computing thesuperimposition region, calibration of the display (2) of the device fora user, an eye distance and a distance between the eyes and the display(2) being determined, the ascertained eye distance and the determineddistance when computing a superimposition region being taken intoaccount in the display (2).
 9. Method according to claim 1, furthercomprising triggering a user notification when a signal of a coupledbiofeedback sensor exceeds a predefined threshold and/or when the facialrecognition detects an affective deflection or resonance above apredefined threshold on a recognized face and/or when voice recognitionrecognizes a voice of an individual that has been stored beforehand as astressor in a database.
 10. Device for changing the affective visualinformation in the field of vision of a user, the device having at leastone image sensor (14) and at least one display (2), as well as aprocessing unit (6) that is connected to the image sensor (14) and thedisplay (2), the image sensor (14) being set up to acquire an image of avisual field of the device, the display (2) being set up to superimposethe field of vision of a user in a superimposition region of the display(2) with alternative image data, wherein charactcrizcd in that theprocessing unit (6) is set up to carry out facial recognition with animage that has been acquired by the image sensor (14), to determinepositions of eyes and mouth of a face that has been recognized in thefacial recognition, and to compute a superimposition region in thedisplay (2) for superimposing the field of vision depending on thedetermined positions of eyes and mouth.
 11. Device according to claim10, wherein charactcrizcd in that the display (2) is a screen or adisplay, in particular a transparent display.
 12. Device according toclaim 11, wherein the device charactcrizcd in that it is augmentedreality glasses (1).
 13. Device according to claim 10, wherein theprocessing unit (6) has facial recognition software.
 14. Deviceaccording to claim 10, further comprising a biofeedback sensor, theprocessing unit (6) being set up and connected to the biofeedback sensorto activate superimposition depending on a signal of the biofeedbacksensor.
 15. Device according to claim 10, further comprising anotification apparatus that is set up to deliver an optical, acoustic orhaptic signal, the processing unit (6) being set up and connected to thenotification apparatus to initiate an optical, acoustic or haptic signalwhen a specific affective pattern and/or an individual assigned to aspecific affect is recognized.
 16. Method according to claim 2, whereinthe superimposition region corresponds essentially to a triangle spreadout between the positions of the two eyes and of the mouth.
 17. Methodaccording to claim 2, wherein the superimposition region encompassesessentially the entire field of view outside of a triangle spread outbetween the positions of the two eyes and the mouth.
 18. Methodaccording to claim 2, wherein the alternative image data correspond toaltered imaging of the visual field of the device in the superimpositionregion, the change in the imaging encompassing a change in brightness, areduction in image resolution, a change in delineation and/or incontrast, color marking and/or manipulation of facial traits.
 19. Methodaccording to claim 3, wherein the alternative image data correspond toaltered imaging of the visual field of the device in the superimpositionregion, the change in the imaging encompassing a change in brightness, areduction in image resolution, a change in delineation and/or incontrast, color marking and/or manipulation of facial traits.
 20. Methodaccording to claim 4, wherein the alternative image data correspond toaltered imaging of the visual field of the device in the superimpositionregion, the change in the imaging encompassing a change in brightness, areduction in image resolution, a change in delineation and/or incontrast, color marking and/or manipulation of facial traits.